Adaptive Dynamic Pruning for Non-IID Federated Learning

Federated Learning~(FL) has emerged as a new paradigm of training machine learning models without sacrificing data security and privacy. Learning models at edge devices such as cell phones is one of the most common use case of FL. However, the limited computing power and energy constraints of edge devices hinder the adoption of FL for both model training and deployment, especially for the resource-hungry Deep Neural Networks~(DNNs). To this end, many model compression methods have been proposed and network pruning is among the most well-known. However, a pruning policy for a given model is highly dataset-dependent, which is not suitable for non-Independent and Identically Distributed~(Non-IID) FL edge devices. In this paper, we present an adaptive pruning scheme for edge devices in an FL system, which applies dataset-aware dynamic pruning for inference acceleration on Non-IID datasets. Our evaluation shows that the proposed method accelerates inference by $2\times$~($50\%$ FLOPs reduction) while maintaining the model's quality on edge devices

Towards Seamless Management of AI Models in High-Performance Computing

With the increasing prevalence of artificial intelligence (AI) in diverse science/engineering communities, AI models emerge on an unprecedented scale among various domains. However, given the complexity and diversity of the software and hardware environments, reusing AI artifacts (models and datasets) is extremely challenging, especially with AI-driven science applications. Building an ecosystem to run and reuse AI applications/datasets at scale efficiently becomes increasingly essential for diverse science and engineering and high-performance computing (HPC) communities. In this paper, we innovate over an HPC-AI ecosystem -- HPCFair, which enables the Findable, Accessible, Interoperable, and Reproducible (FAIR) principles. HPCFair enables the collection of AI models/datasets allowing users to download/upload AI artifacts with authentications. Most importantly, our proposed framework provides user-friendly APIs for users to easily run inference jobs and customize AI artifacts to their tasks as needed. Our results show that, with HPCFair API, users irrespective of technical expertise in AI, can easily leverage AI artifacts to their tasks with minimal effort.Comment: Accepted at the 2nd Annual AAAI Workshop on AI to Accelerate Science and Engineering (AI2ASE

A ribosomally synthesised and post-translationally modified peptide containing a β-enamino acid and a macrocyclic motif

Ribosomally synthesized and post-translationally modified peptides (RiPPs) are structurally complex natural products with diverse bioactivities. Here we report discovery of a RiPP, kintamdin, for which the structure is determined through spectroscopy, spectrometry and genomic analysis to feature a bis-thioether macrocyclic ring and a β-enamino acid residue. Biosynthetic investigation demonstrated that its pathway relies on four dedicated proteins: phosphotransferase KinD, Lyase KinC, kinase homolog KinH and flavoprotein KinI, which share low homologues to enzymes known in other RiPP biosynthesis. During the posttranslational modifications, KinCD is responsible for the formation of the characteristic dehydroamino acid residues including the β-enamino acid residue, followed by oxidative decarboxylation on the C-terminal Cys and subsequent cyclization to provide the bis-thioether ring moiety mediated by coordinated action of KinH and KinI. Finally, conserved genomic investigation allows further identification of two kintamdin-like peptides among the kin-like BGCs, suggesting the occurrence of RiPPs from actinobacteria

Topology-aware efficient and transferable model compression using graph representation and reinforcement learning

Deep neural networks (DNNs) have found widespread applications across many domains. However, deploying these models on devices with limited computational and storage capabilities, like mobile devices, poses significant challenges. Model compression, aiming to make these large models more efficient without significant performance loss, is an active research area. However, traditional model compression techniques often require expert knowledge and overlook the inherent structural information within DNNs. To address these challenges, this thesis proposes two novel techniques, Auto Graph Encoder-decoder Model Compression (AGMC) and Graph Neural Network with Reinforcement Learning (GNN-RL). AGMC and GNN-RL harness the power of graph neural networks (GNNs) and reinforcement learning (RL) to extract structural information from DNNs, modeled as computational graphs, and automatically derive efficient compression policies. These policies are then used to guide the model compression process, resulting in compact yet effective DNNs. AGMC combines a GNN-based DNN embedding mechanism with RL to learn and apply effective compression strategies. The results showcase the superiority of AGMC over traditional rule-based DNN embedding techniques, yielding improved performance and higher compression ratios. It outperforms both handcrafted and learning-based model compression approaches on over-parameterized and mobile-friendly DNNs. On over-parameterized DNNs like ResNet-56, our method surpasses previous state-of-the-art methods with higher accuracy. Furthermore, on compact DNNs like MobileNet-v2, AGMC achieves a higher compression ratio with minimal accuracy loss. GNN-RL extends this work by introducing a novel multi-stage graph embedding technique to capture DNN topologies, along with RL to determine an optimal compression policy. The effectiveness of GNN-RL is demonstrated on a diverse set of DNNs, including the ResNet family, VGG-16, MobileNet-v1/v2, and ShuffleNet. GNN-RL achieved competitive results, providing higher compression ratios with less fine-tuning, significantly reducing the computational resources required while maintaining outstanding model performance. These methods pave the way for more automated and efficient model compression, enabling the deployment of complex DNNs on resource-constrained devices

Topology-aware efficient and transferable model compression using graph representation and reinforcement learning

Deep neural networks (DNNs) have found widespread applications across many domains. However, deploying these models on devices with limited computational and storage capabilities, like mobile devices, poses significant challenges. Model compression, aiming to make these large models more efficient without significant performance loss, is an active research area. However, traditional model compression techniques often require expert knowledge and overlook the inherent structural information within DNNs. To address these challenges, this thesis proposes two novel techniques, Auto Graph Encoder-decoder Model Compression (AGMC) and Graph Neural Network with Reinforcement Learning (GNN-RL). AGMC and GNN-RL harness the power of graph neural networks (GNNs) and reinforcement learning (RL) to extract structural information from DNNs, modeled as computational graphs, and automatically derive efficient compression policies. These policies are then used to guide the model compression process, resulting in compact yet effective DNNs. AGMC combines a GNN-based DNN embedding mechanism with RL to learn and apply effective compression strategies. The results showcase the superiority of AGMC over traditional rule-based DNN embedding techniques, yielding improved performance and higher compression ratios. It outperforms both handcrafted and learning-based model compression approaches on over-parameterized and mobile-friendly DNNs. On over-parameterized DNNs like ResNet-56, our method surpasses previous state-of-the-art methods with higher accuracy. Furthermore, on compact DNNs like MobileNet-v2, AGMC achieves a higher compression ratio with minimal accuracy loss. GNN-RL extends this work by introducing a novel multi-stage graph embedding technique to capture DNN topologies, along with RL to determine an optimal compression policy. The effectiveness of GNN-RL is demonstrated on a diverse set of DNNs, including the ResNet family, VGG-16, MobileNet-v1/v2, and ShuffleNet. GNN-RL achieved competitive results, providing higher compression ratios with less fine-tuning, significantly reducing the computational resources required while maintaining outstanding model performance. These methods pave the way for more automated and efficient model compression, enabling the deployment of complex DNNs on resource-constrained devices

Topology-aware efficient and transferable model compression using graph representation and reinforcement learning

Deep neural networks (DNNs) have found widespread applications across many domains. However, deploying these models on devices with limited computational and storage capabilities, like mobile devices, poses significant challenges. Model compression, aiming to make these large models more efficient without significant performance loss, is an active research area. However, traditional model compression techniques often require expert knowledge and overlook the inherent structural information within DNNs. To address these challenges, this thesis proposes two novel techniques, Auto Graph Encoder-decoder Model Compression (AGMC) and Graph Neural Network with Reinforcement Learning (GNN-RL). AGMC and GNN-RL harness the power of graph neural networks (GNNs) and reinforcement learning (RL) to extract structural information from DNNs, modeled as computational graphs, and automatically derive efficient compression policies. These policies are then used to guide the model compression process, resulting in compact yet effective DNNs. AGMC combines a GNN-based DNN embedding mechanism with RL to learn and apply effective compression strategies. The results showcase the superiority of AGMC over traditional rule-based DNN embedding techniques, yielding improved performance and higher compression ratios. It outperforms both handcrafted and learning-based model compression approaches on over-parameterized and mobile-friendly DNNs. On over-parameterized DNNs like ResNet-56, our method surpasses previous state-of-the-art methods with higher accuracy. Furthermore, on compact DNNs like MobileNet-v2, AGMC achieves a higher compression ratio with minimal accuracy loss. GNN-RL extends this work by introducing a novel multi-stage graph embedding technique to capture DNN topologies, along with RL to determine an optimal compression policy. The effectiveness of GNN-RL is demonstrated on a diverse set of DNNs, including the ResNet family, VGG-16, MobileNet-v1/v2, and ShuffleNet. GNN-RL achieved competitive results, providing higher compression ratios with less fine-tuning, significantly reducing the computational resources required while maintaining outstanding model performance. These methods pave the way for more automated and efficient model compression, enabling the deployment of complex DNNs on resource-constrained devices

Topology-aware efficient and transferable model compression using graph representation and reinforcement learning

Deep neural networks (DNNs) have found widespread applications across many domains. However, deploying these models on devices with limited computational and storage capabilities, like mobile devices, poses significant challenges. Model compression, aiming to make these large models more efficient without significant performance loss, is an active research area. However, traditional model compression techniques often require expert knowledge and overlook the inherent structural information within DNNs. To address these challenges, this thesis proposes two novel techniques, Auto Graph Encoder-decoder Model Compression (AGMC) and Graph Neural Network with Reinforcement Learning (GNN-RL). AGMC and GNN-RL harness the power of graph neural networks (GNNs) and reinforcement learning (RL) to extract structural information from DNNs, modeled as computational graphs, and automatically derive efficient compression policies. These policies are then used to guide the model compression process, resulting in compact yet effective DNNs. AGMC combines a GNN-based DNN embedding mechanism with RL to learn and apply effective compression strategies. The results showcase the superiority of AGMC over traditional rule-based DNN embedding techniques, yielding improved performance and higher compression ratios. It outperforms both handcrafted and learning-based model compression approaches on over-parameterized and mobile-friendly DNNs. On over-parameterized DNNs like ResNet-56, our method surpasses previous state-of-the-art methods with higher accuracy. Furthermore, on compact DNNs like MobileNet-v2, AGMC achieves a higher compression ratio with minimal accuracy loss. GNN-RL extends this work by introducing a novel multi-stage graph embedding technique to capture DNN topologies, along with RL to determine an optimal compression policy. The effectiveness of GNN-RL is demonstrated on a diverse set of DNNs, including the ResNet family, VGG-16, MobileNet-v1/v2, and ShuffleNet. GNN-RL achieved competitive results, providing higher compression ratios with less fine-tuning, significantly reducing the computational resources required while maintaining outstanding model performance. These methods pave the way for more automated and efficient model compression, enabling the deployment of complex DNNs on resource-constrained devices

Topology-aware efficient and transferable model compression using graph representation and reinforcement learning

Deep neural networks (DNNs) have found widespread applications across many domains. However, deploying these models on devices with limited computational and storage capabilities, like mobile devices, poses significant challenges. Model compression, aiming to make these large models more efficient without significant performance loss, is an active research area. However, traditional model compression techniques often require expert knowledge and overlook the inherent structural information within DNNs. To address these challenges, this thesis proposes two novel techniques, Auto Graph Encoder-decoder Model Compression (AGMC) and Graph Neural Network with Reinforcement Learning (GNN-RL). AGMC and GNN-RL harness the power of graph neural networks (GNNs) and reinforcement learning (RL) to extract structural information from DNNs, modeled as computational graphs, and automatically derive efficient compression policies. These policies are then used to guide the model compression process, resulting in compact yet effective DNNs. AGMC combines a GNN-based DNN embedding mechanism with RL to learn and apply effective compression strategies. The results showcase the superiority of AGMC over traditional rule-based DNN embedding techniques, yielding improved performance and higher compression ratios. It outperforms both handcrafted and learning-based model compression approaches on over-parameterized and mobile-friendly DNNs. On over-parameterized DNNs like ResNet-56, our method surpasses previous state-of-the-art methods with higher accuracy. Furthermore, on compact DNNs like MobileNet-v2, AGMC achieves a higher compression ratio with minimal accuracy loss. GNN-RL extends this work by introducing a novel multi-stage graph embedding technique to capture DNN topologies, along with RL to determine an optimal compression policy. The effectiveness of GNN-RL is demonstrated on a diverse set of DNNs, including the ResNet family, VGG-16, MobileNet-v1/v2, and ShuffleNet. GNN-RL achieved competitive results, providing higher compression ratios with less fine-tuning, significantly reducing the computational resources required while maintaining outstanding model performance. These methods pave the way for more automated and efficient model compression, enabling the deployment of complex DNNs on resource-constrained devices

Topology-aware efficient and transferable model compression using graph representation and reinforcement learning

Deep neural networks (DNNs) have found widespread applications across many domains. However, deploying these models on devices with limited computational and storage capabilities, like mobile devices, poses significant challenges. Model compression, aiming to make these large models more efficient without significant performance loss, is an active research area. However, traditional model compression techniques often require expert knowledge and overlook the inherent structural information within DNNs. To address these challenges, this thesis proposes two novel techniques, Auto Graph Encoder-decoder Model Compression (AGMC) and Graph Neural Network with Reinforcement Learning (GNN-RL). AGMC and GNN-RL harness the power of graph neural networks (GNNs) and reinforcement learning (RL) to extract structural information from DNNs, modeled as computational graphs, and automatically derive efficient compression policies. These policies are then used to guide the model compression process, resulting in compact yet effective DNNs. AGMC combines a GNN-based DNN embedding mechanism with RL to learn and apply effective compression strategies. The results showcase the superiority of AGMC over traditional rule-based DNN embedding techniques, yielding improved performance and higher compression ratios. It outperforms both handcrafted and learning-based model compression approaches on over-parameterized and mobile-friendly DNNs. On over-parameterized DNNs like ResNet-56, our method surpasses previous state-of-the-art methods with higher accuracy. Furthermore, on compact DNNs like MobileNet-v2, AGMC achieves a higher compression ratio with minimal accuracy loss. GNN-RL extends this work by introducing a novel multi-stage graph embedding technique to capture DNN topologies, along with RL to determine an optimal compression policy. The effectiveness of GNN-RL is demonstrated on a diverse set of DNNs, including the ResNet family, VGG-16, MobileNet-v1/v2, and ShuffleNet. GNN-RL achieved competitive results, providing higher compression ratios with less fine-tuning, significantly reducing the computational resources required while maintaining outstanding model performance. These methods pave the way for more automated and efficient model compression, enabling the deployment of complex DNNs on resource-constrained devices

Topology-aware efficient and transferable model compression using graph representation and reinforcement learning

Deep neural networks (DNNs) have found widespread applications across many domains. However, deploying these models on devices with limited computational and storage capabilities, like mobile devices, poses significant challenges. Model compression, aiming to make these large models more efficient without significant performance loss, is an active research area. However, traditional model compression techniques often require expert knowledge and overlook the inherent structural information within DNNs. To address these challenges, this thesis proposes two novel techniques, Auto Graph Encoder-decoder Model Compression (AGMC) and Graph Neural Network with Reinforcement Learning (GNN-RL). AGMC and GNN-RL harness the power of graph neural networks (GNNs) and reinforcement learning (RL) to extract structural information from DNNs, modeled as computational graphs, and automatically derive efficient compression policies. These policies are then used to guide the model compression process, resulting in compact yet effective DNNs. AGMC combines a GNN-based DNN embedding mechanism with RL to learn and apply effective compression strategies. The results showcase the superiority of AGMC over traditional rule-based DNN embedding techniques, yielding improved performance and higher compression ratios. It outperforms both handcrafted and learning-based model compression approaches on over-parameterized and mobile-friendly DNNs. On over-parameterized DNNs like ResNet-56, our method surpasses previous state-of-the-art methods with higher accuracy. Furthermore, on compact DNNs like MobileNet-v2, AGMC achieves a higher compression ratio with minimal accuracy loss. GNN-RL extends this work by introducing a novel multi-stage graph embedding technique to capture DNN topologies, along with RL to determine an optimal compression policy. The effectiveness of GNN-RL is demonstrated on a diverse set of DNNs, including the ResNet family, VGG-16, MobileNet-v1/v2, and ShuffleNet. GNN-RL achieved competitive results, providing higher compression ratios with less fine-tuning, significantly reducing the computational resources required while maintaining outstanding model performance. These methods pave the way for more automated and efficient model compression, enabling the deployment of complex DNNs on resource-constrained devices